Robust Character Labeling in Movie Videos: Data Resources and Self-Supervised Feature Adaptation

Robust face clustering is a vital step in enabling computational understanding of visual character portrayal media. Face for long-form content challenging because variations appearance and lack supporting large-scale labeled data. Our work this paper focuses on two key aspects problem: the domain-specific training or benchmark datasets, adapting embeddings learned web images to content, specifically movies. First, we present dataset over 169,000 tracks curated from 240 Hollywood movies with weak labels whether pair belong same different character. We propose an offline algorithm based nearest-neighbor search embedding space mine hard-examples these tracks. then investigate triplet-loss multiview correlation-based methods hard-examples. experimental results highlight usefulness weakly data feature adaptation. Overall, find that adaptation yields more discriminative robust embeddings. Its performance downstream verification tasks comparable state-of-the-art domain. also SAIL-Movie Character Benchmark corpus developed augment existing benchmarks. It consists racially diverse actors provides face-quality subsequent error analysis. hope datasets can further advance automatic labeling videos. All resources are available freely at https://sail.usc.edu/~ccmi/multiface.